Searching For Applications Based On Application Usage

ABSTRACT

A method includes retrieving, at computing devices, event analytics data for a software application installed on user devices. The event analytics data indicates an amount of time each accessible application state was accessed by the user devices. The method includes mapping function names indicating functionality to the application states. The method includes determining an amount of time associated with application states corresponding to each function name. The method includes determining a usage fraction for each of the function names indicating the amount of time associated with the function name relative to the rest of the function names. The method includes assigning function keywords to an application search record associated with the software application based on the usage fractions. The method includes, in response to a search query, identifying and transmitting an application search record based on a match between terms of the search query and terms of the function keywords.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/985,605 filed on Dec. 31, 2015 and U.S. patent applicationSer. No. 14/985,667 filed on Dec. 31, 2015. The entire disclosures ofthe applications referenced above are incorporated by reference.

FIELD

This disclosure relates to techniques for searching, and moreparticularly to performing searches for applications based onapplication usage.

BACKGROUND

In recent years, use of computers, smartphones, and otherInternet-connected devices has grown exponentially. Correspondingly, thenumber of available software applications for such devices has alsogrown. Today, many diverse native and web software applications can beaccessed on any number of different devices, including, but not limitedto, smartphones, personal computers, automobiles, and televisions. Thesediverse applications can range from business driven applications, games,educational applications, news applications, shopping applications,messaging applications, media streaming applications, social networkingapplications, and much more. Furthermore, application developers developvast amounts of applications within each genre and each application mayhave numerous editions. As a result, users of these Internet-connecteddevices have encountered the problem of finding the correct native orweb software application offering the information and/or functionalitythat they seek.

SUMMARY

In one example, the present disclosure is directed to a methodcomprising retrieving event analytics data for a software applicationinstalled on a plurality of user devices. The software applicationincludes a plurality of accessible application states. The eventanalytics data indicates a number of times each of the accessibleapplication states were accessed by the user devices. The method furthercomprises mapping function names to the application states accessed bythe user devices. Each of the function names indicates a functionalityassociated with the software application. The method further comprisesdetermining a usage fraction value for each of the function names of thesoftware application based on the number of times each of the functionnames was mapped, the usage fraction value for each function nameindicating the number of times the function name was mapped relative tothe number of times each of the function names was mapped. The methodfurther comprises assigning function keywords to an application searchrecord associated with the software application based on the usagefraction values associated with the function names of the softwareapplication. Additionally, the method comprises receiving a search queryfrom a querying user device and identifying the application searchrecord based on a match between terms of the search query and terms ofthe function keywords assigned to the application search record. Themethod further comprises generating search results based on theidentified application search record and transmitting the search resultsto the querying user device.

In another example, the present disclosure is directed to a systemcomprising one or more computing devices configured to retrieve eventanalytics data for a software application installed on a plurality ofuser devices. The software application includes a plurality ofaccessible application states. The event analytics data indicates anumber of times each of the accessible application states were accessedby the user devices. The one or more computing devices are configured tomap function names to the application states accessed by the userdevices. Each of the function names indicates a functionality associatedwith the software application. The one or more computing devices areconfigured to determine a usage fraction value for each of the functionnames of the software application based on the number of times each ofthe function names was mapped, the usage fraction value for eachfunction name indicating the number of times the function name wasmapped relative to the number of times each of the function names wasmapped. The one or more computing devices are configured to assignfunction keywords to an application search record associated with thesoftware application based on the usage fraction values associated withthe function names of the software application. Additionally, the one ormore computing devices are configured to receive a search query from aquerying user device, identify the application search record based on amatch between terms of the search query and terms of the functionkeywords assigned to the application search record, generate searchresults based on the identified application search record, and transmitthe search results to the querying user device.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more examples are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

FIG. 1 illustrates an example environment including a usage fractiondetermination system and an application search system of the presentdisclosure.

FIG. 2 is an example functional block diagram of the environment of FIG.1.

FIG. 3 is a flow diagram illustrating an example method describing theoperation of the environment of FIG. 1.

FIGS. 4A-4C illustrate example state graphs and function graphs.

FIG. 5 is a functional block diagram of an example usage fractiondetermination system and an example application search system.

FIG. 6 illustrates an example analytics record.

FIG. 7 illustrates an example function mapping module that retrievesevent analytics data and generates function analytics data.

FIGS. 8-9 are flow diagrams illustrating example methods for generatingtime-based usage fraction values and event-based usage fraction values.

FIG. 10 is a flow diagram illustrating example operations performed by ausage fraction determination system.

FIG. 11 illustrates an example user device in communication with anapplication search system.

FIGS. 12A-12B illustrate example application search records.

FIG. 13 is a flow diagram illustrating a method for assigning functionnames and function metadata to an application search record.

FIG. 14 illustrates an example application search module.

FIG. 15 is a flow diagram illustrating an example method for performinga search for applications.

DETAILED DESCRIPTION

The figures and the following description relate to exampleimplementations by way of illustration only. It should be noted thatfrom the following discussion, alternative implementations of thestructures and methods disclosed herein will be readily recognized asviable alternatives that may be employed without departing from thescope of the disclosure.

An application search system (hereinafter “search system”) of thepresent disclosure receives a search query from a user device andgenerates search results including a list of links forpreviewing/downloading applications from a digital distributionplatform. The search system stores a plurality of application searchrecords (hereinafter “search records”) used during the search. Eachsearch record may correspond to a single application and may includeapplication attributes that are descriptive of the applicationassociated with the search record. The search system can identify andscore the search records based on matches between the search query andterms included in the application attributes along with other scoringfeatures. The search system can then generate search results including alist of links retrieved from the scored search records. For example, thesearch results may include link data (e.g., text/images) for renderingthe search results on a user device and application download addressesfor previewing and/or downloading the applications.

The application attributes of the search records can include functionnames and/or function metadata (referred to herein generally as“function keywords”) along with a variety of other data (e.g., anapplication description and application statistics). The function namesand function metadata may include terms that describe the variousfunctions of the application. For example an application directedmapping and navigation may provide functions (e.g., function names),such as “Find transportation,” “Driving directions,” and “Walkingdirections.” Each of the function names may be associated with functionmetadata. For example, the “Driving directions” function may beassociated with terms such as “map,” “car,” “traffic,” “construction,”and “shortest.” The search system may assign the function names andfunction metadata to the search records based on how users engage withthe applications (e.g., based on which states of the applications areaccessed). The search system can identify and score the search recordsduring a search based on the function names and/or the functionmetadata.

A usage fraction determination system of the present disclosuredetermines usage fraction values for applications that indicate howusers engage with the applications' different functionalities. Thesearch system may use the usage fraction values to determine how toassign function metadata to the search records. For example, the searchsystem may assign function names and function metadata to a searchrecord based on which functions of the applications are used, asindicated by the usage fraction values. The additional function namesand function metadata in the search records may help to surface thesearch records during search because the function names and functionmetadata may provide keyword matches to search queries that may nototherwise have occurred based on the other application attributes.Additionally, the additional description provided by the function namesand function metadata may improve relevance by providing relevantkeywords for matching with the search query. In some implementations,the usage fraction values associated with function names and functionmetadata may be used by the search system in order to improve relevance.

Applications provide a variety of different functions for users. In thepresent disclosure, each application (e.g., native/web application) isassociated with a set of functions (e.g., a set of function names). Forexample, a review application that provides reviews of businesses mayhave a plurality of functions, such as “Read Restaurant Review,” “ReadStore Review,” “Provide Restaurant Info,” “Show menu,” and “Show foodphotos.” As another example, a mapping and navigation application mayhave a plurality of functions, such as “Find transportation tolocation,” “Driving Directions,” “Walking Directions,” and “Show nearbygas stations.” Each function of an application can be associated with(e.g., assigned to) one or more states of the application. As describedherein, each function name can be descriptive of the states with whichthe function is associated. For example, the review applicationdescribed above may include a plurality of application states that showmenu items for different restaurants. In this example, each of theapplication states that show menu items may be associated with the “Showmenu” function.

The usage fraction determination system determines the usage fractionvalues based on analytics data describing how users use the applications(e.g., which application states are accessed). A usage fraction valueassociated with a function may indicate the relative usage of thefunction among the plurality of other functions provided by theapplication. In some cases, usage fraction values for the same/similarfunctions from a plurality of different applications can be used tocompare the functions across the plurality of different applications.For example, those functions that have higher usage fraction values anduser engagement may be more relevant than those functions that havelower usage fraction values and/or less user engagement. Informationyielded by the usage fraction values, along with other information, canbe used in a variety of different ways. In some implementations, asearch system of the present disclosure may leverage the usage fractionvalues to generate relevant search results.

Each function of an application can be assigned a usage fraction valueindicating how much the function is used (e.g., across a group ofusers). For example, for a set of functions, each of the functions canbe assigned a usage fraction value indicating the respective amount ofusage for each function. In a more specific example, if an applicationhas four functions, each of the four functions can be assigned arespective usage fraction value. The usage fraction values for anapplication may be percentage values that indicate an amount of usage ofa single function relative to the other functions of the application.The usage fraction values may add up to 100 percent.

The usage fraction determination system may determine different types ofusage fraction values for the same application. Example usage fractiontypes may include a time-based usage fraction type and an event-basedusage fraction type. An application may have a plurality of usagefraction values for each type of usage fraction. For example, for asingle usage fraction type, an application may have a single usagefraction value for each of the functions of the application. In aspecific example, if an application has four functions and two usagefraction types, the application may have two sets of usage fractionvalues, one set of four usage fraction values for a first type of usagefraction and another set of four usage fraction values for a second typeof usage fraction. In this example, each function of the application canbe assigned two usage fraction values.

In some implementations, the usage fraction determination system maydetermine one or more user engagement values that indicate an amount ofusers (e.g., number of user devices) that engaged with the application(e.g., the popularity of the application). For example, a userengagement value may indicate the number of users for which analyticsdata was collected (e.g., the analytics data upon which the usagefraction values are based). Additionally, or alternatively, userengagement values may include, but are not limited to, a number ofapplication downloads, a number of daily active users, and a number ofmonthly active users. In some implementations, a search system of thepresent disclosure may leverage both usage fraction values and userengagement values to generate relevant search results (e.g., using anormalized usage fraction value).

FIGS. 1-10 are directed to determining usage fraction values and userengagement values that may be used by the search system to assignfunction keywords to search records. FIG. 11 illustrates an example userdevice searching for applications. FIGS. 12A-12B illustrate examplesearch records including function names, function metadata, usagefraction values, user engagement values, and other data. FIG. 13illustrates an example method for assigning function names, functionmetadata, and other data to a search record. FIGS. 14-15 describeoperation of an example application search module that generatesapplication search results based on the function names, functionmetadata, and other data of the search records.

FIG. 1 illustrates an example environment that includes one or moreapplication developer systems 100, one or more analytics providersystems 102, and one or more digital distribution platforms 104. Theapplication developer systems 100 represent computing systems of nativeapplication developers. The native application developers develop nativeapplications that execute on user devices 106 and then provide thenative applications for download on the digital distribution platforms104. The digital distribution platforms 104 may represent computingsystems that are configured to distribute native applications to userdevices 106. Example digital distribution platforms 104 include, but arenot limited to, the GOOGLE PLAY® digital distribution platform byGoogle, Inc. and the APP STORE® digital distribution platform by Apple,Inc. In some cases, the same party may own a digital distributionplatform 104 and the application search system 110. In this case, thesame party may operate the application search system 110 as a searchsystem for the digital distribution platform 104.

The analytics provider systems 102 represent computing systems that maybe operated by analytics software developers. The analytics softwaredevelopers may develop analytics modules (e.g., application analyticsmodules 212-1 and/or device analytics modules 212-2 of FIG. 2) andprovide the analytics modules for inclusion in native applications andin user devices 106. For example, the analytics software developers mayprovide the analytics modules for download on the analytics providersystems 102.

FIG. 1 includes a usage fraction determination system 108 (hereinafter“UF determination system 108”) and an application search system 110(hereinafter “search system 110”). The UF determination system 108retrieves the analytics data from the application developers systems 100and/or the analytics provider systems 102 and determines usage fractionvalues based on the retrieved analytics data. In some implementations,the UF determination system 108 may also determine user engagementvalues. The search system 110 assigns function names and functionmetadata to search records (e.g., illustrated at 1204-1, 1204-2 in FIGS.12A-12B) based on the usage fraction values associated with variousfunctions of the applications. The search system 110 receives searchqueries from user devices 106 and generates search results based on theassigned function names and function metadata, and then transmits thesearch results to the user devices 106.

The user devices 106, application developer systems 100, analyticsprovider systems 102, digital distribution platforms 104, UFdetermination system 108, and search system 110 may communicate via anetwork 112. The network 112 may include various types of networks, suchas a local area network (LAN), wide area network (WAN), and/or theInternet.

User devices 106 include, but are not limited to, smart phones, tabletcomputers, laptop computers, and desktop computers. User devices 106 mayalso include other computing devices having other form factors, such ascomputing devices included in vehicles, gaming devices, televisions, orother appliances (e.g., networked home automation devices and homeappliances). The user devices 106 may use a variety of differentoperating systems (illustrated at 1112 in FIG. 11). In an example wherea user device 106 is a mobile device, the user device 106 may run anoperating system including, but not limited to, ANDROID® developed byGoogle Inc. or IOS® developed by Apple Inc. In an example where a userdevice 106 is a laptop or desktop computing device, the user device 106may run an operating system including, but not limited to, MICROSOFTWINDOWS® by Microsoft Corporation, MAC OS® by Apple, Inc., or Linux.

User devices 106 may have a plurality of installed native applications(illustrated at 1120 in FIG. 11). In some examples, native applicationsmay function without communication with other remote computing devices(e.g., via the Internet). In other examples, a native applicationinstalled on a user device may access information from other remotecomputing devices during operation. For example, a weather applicationinstalled on a user device may access the latest weather information viathe Internet and display the accessed weather information to the userthrough the installed weather application. In still other examples, anapplication (e.g., a web based application) may be partially executed bythe user's computing device and partially executed by a remote computingdevice. For example, a web application may be an application that isexecuted, at least in part, by a web server and accessed by a webbrowser on the user device (e.g., a native installed web browserapplication illustrated at 1102 in FIG. 11). Example web applicationsmay include, but are not limited to, web-based email, online auctions,and online retail sites.

FIG. 2 illustrates a functional block diagram including the systems ofFIG. 1. FIG. 3 illustrates a method describing operation of the systemsincluded in FIGS. 1-2. Example communication between the systems anduser devices are labeled with callouts 200-206. The callouts 200-206illustrate an example sequence of communications between the systems anduser devices. The method of FIG. 3 also includes callouts 200-206 thatcorrespond to the callouts of FIG. 2. Example operations performed bythe systems are now described with respect to FIGS. 2-3.

Referring to FIG. 3, the analytics providers develop analytics modules210 and provide the analytics modules 212 to the application developersystems 100 at 200. For example, the analytics providers may provideapplication analytics modules 212-1 (AAM 212-1 in FIG. 2) to thedevelopers for download on the analytics provider systems 102. In thisexample, the application developer systems 100 may retrieve theapplication analytics modules 212-1 from the analytics provider systems102 and include the application analytics modules 212-1 in the nativeapplications 210 under development. Although the analytics developersmay provide application analytics modules 212-1 to the applicationdevelopers for insertion into native applications 210, in someimplementations, analytics provider systems 102 can provide deviceanalytics modules 212-2 (DAM 212-2 in FIG. 2) for inclusion on the oneor more user devices 106 outside of native applications 210 provided onthe digital distribution platforms 104.

At 201, the application developers incorporate the application analyticsmodules 212-1 into the native applications 210 and upload the nativeapplications 210 to the digital distribution platforms 104. At 202, aplurality of user devices 106 download and install the nativeapplications 210 including the application analytics modules 212-1. At203, users use the native applications 210 on the user devices 106 andthe application analytics modules 212-1 acquire analytics data (e.g.,event analytics data) associated with usage of the applications. Theapplication analytics modules 212-1 transfer the acquired analytics datato the respective analytics provider systems 102 and/or the respectiveapplication developer systems 100. In some cases, the applicationdeveloper systems 100 can store the analytics data. In other cases, theanalytics provider systems 102 may store the analytics data and allowthe application developer systems 100 to access the analytics data, asindicated by the analytics services provided at 207. Example analyticsservices may include formatting analytics data, analyzing the analyticsdata, displaying analytics data (e.g., via a dashboard), and providingthe analytics data to developers for download.

At 204, the UF determination system 108 retrieves the analytics datafrom the application developer systems 100 and/or the analytics providersystems 102, depending on which systems store the analytics dataacquired by the user devices 106 (e.g., application analytics modules212-1 and/or device analytics modules 212-2). At 205, the UFdetermination system 108 determines usage fraction values based on theretrieved analytics data and the search system 205 assigns functionnames and function metadata to search records based on the usagefraction values. At 206, the search system 110 receives search queriesfrom the user devices 106, generates search results based on thefunction names and/or function metadata, and then transmits the searchresults to the user devices 106.

FIGS. 4A-4C illustrate example application state/function graphsincluding application states, application functions, and transitionsbetween the application states and application functions. Applicationstates, functions, and transitions are described hereinafter withrespect to FIGS. 4A-4C.

An application state can refer to a page/screen within an application.In some cases, an application state can be reached as a result of userinteraction with the user device on which the native application isinstalled (e.g., selection of a user selectable link). A user device canaccess an application state using an access mechanism (e.g., a URL).Access mechanisms may include at least one of a native applicationaccess mechanism (hereinafter “application access mechanism”) and a webaccess mechanism. The user device 106 may use the access mechanisms toaccess functionality of applications.

An application access mechanism may be a string that includes areference to a native application (e.g., one of native applications 210installed on the user devices 106) and indicates one or more operationsfor the user device 106 to perform. If a user selects a user selectablelink including an application access mechanism, the user device 106 maylaunch the native application referenced in the application accessmechanism and set the application into a state specified by theapplication access mechanism. In some cases the application accessmechanisms can include application resource identifiers (e.g., definedby the application developers). Such application resource identifiersmay have a format similar to a web URL schema including a domain and apath. For example, an application resource identifier for the OPENTABLE®application developed by OpenTable, Inc. may be“vnd.opentable.deeplink://opentable.com/restaurant/profile?rid=1180&refid=1.”This specific application resource identifier may cause the OPENTABLE®Android native application to launch a restaurant table reservation pagefor a restaurant named “The French Laundry.” In some cases, anapplication access mechanism may include operations for the user device106 to perform (e.g., a script) in addition to, or in place of, theapplication resource identifiers. Example operations may include, butare not limited to, launching a native application, waiting for thenative application to start, creating and sending a search request to aserver, setting a current geo-location in a native application, making arestaurant reservation, sending a text message, and adding anappointment to a calendar.

A web access mechanism may include a resource identifier that includes areference to a web resource (e.g., a page of a web application/website).For example, a web access mechanism may include a uniform resourcelocator (URL) (i.e., a web address) used with hypertext transferprotocol (HTTP). If a user selects a user selectable link including aweb access mechanism, the user device 106 may launch the web browserapplication (e.g., illustrated at 1102 in FIG. 11) and retrieve the webresource indicated in the resource identifier. Put another way, if auser selects a user selectable link including a web access mechanism,the user device 106 may launch the web browser application and access astate (e.g., a page) of a web application/website. In some examples, webaccess mechanisms may include URLs for mobile-optimized sites and/orfull sites.

A user device 106 can access native application states in a variety ofdifferent ways from within the native application or from outside of thenative application. In one example, a user can select a search resultlink (associated with an access mechanism) on a webpage or a searchresult link within a native search application that causes a nativeapplication to launch to an application home page or into another stateof the native application. In another example, a user can select a weblink (e.g., a web page hyperlink) that causes a native application to beset into an application state specified by the web link. In anotherexample, a user can select a link within a native application that setsthe native application to a different state within the same nativeapplication or accesses a state within a different native application.Although the user device 106 may launch application states in responseto the selection of links, in other examples, other types of userinteractions with other GUI elements (within our outside of the nativeapplication) may cause the user device to access application states.Example interactions may include user interaction with other GUIelements within/outside of the application, such as GUI buttons (e.g.,an OK button, a back button, etc.), menus (e.g., drop down menus), orother navigational GUI elements.

A user can interact with the user device 106 to transition from oneapplication state to another. For example, a user can transition from afirst state of a first application to a second state of the firstapplication via a user interaction with the first state (e.g., selectionof a link in the first state or interaction with a GUI element of thefirst state). As another example, a user can transition from a state ofa first application to a state of a second application via a userinteraction with the state of the first application (e.g., via selectionof a link in the state of the first application or interaction with aGUI element of the state). Since different states may include differentpages/screens, at least a portion (or all) of the page/screen may changeduring a transition between states.

Application states may differ among different applications. For example,application states may display information retrieved from the internet(or the user device 106), provide interactive GUI elements, play media(audio/video), and provide other features. The number of applicationstates for a single application can vary, depending on the application.In some cases, applications can have thousands or millions of differentstates. An example application that provides restaurant reviews can haveone or more states for each restaurant. Similarly, an exampleapplication that provides movie reviews can have one or more states foreach movie, actor, and director.

FIGS. 4A-4B illustrate an example application state graph thatrepresents an application's states (nodes labeled S#) and transitions(labeled T#) between the states. The transitions between applicationstates are represented by arrows. Double ended arrows may indicate thatthe user can go back and forth between states (e.g., using the GUI). Forexample, a user can navigate back and forth between states S1 and S2.Single ended arrows may indicate that a user can transition from a firststate to a second state, but may not directly navigate back to the firststate from within the second state (e.g., via the GUI). For example, inFIG. 4A, a user may transition from state S7 to state S3, but may notaccess state S7 directly from state S3. FIG. 4A may only show a portionof the application states available in the application. Accordingly, thepresence of additional application states is indicated in FIG. 4A.

FIG. 4B illustrates example transitions T1-T5 between states S1 and S2.As described above, the specific transitions in both directions can becondensed into a single double ended arrow, as illustrated in FIG. 4A. Auser may transition between states in a variety of different manners.The transitions T1-T5 may represent any of a variety of ways totransition between states in an application, such as selecting a link,pushing a GUI button, and/or making a menu selection.

The analytics modules 212 can log various events described herein. Insome implementations, the analytics modules 212 can log a user accessingstates of an application, such as when a user enters a state, leaves astate (e.g., closes the application/window and/or powers off thedevice), and transitions from one state to another. For example, theanalytics modules 212 may log access mechanisms (e.g., accessed URLsand/or selected web/app links) that are indicative of thestates/functions a user has accessed. In some implementations, theanalytics modules 212 may log a user's interactions (e.g., clicks) withvarious GUI elements that indicate which states/functions the user hasaccessed. Example GUI elements may include, but are not limited to, GUIbuttons (e.g., an OK button, a back button, etc.), menus (e.g., dropdown menus), and other navigational GUI elements. In someimplementations, the analytics modules 212 may log application specificand/or OS specific operations indicating the states/functions accessedby the user. For example, the analytics modules 212 may log Androidintents/activities that are indicative of states/functions. In someimplementations, the analytics modules 212 may log when a user views astate of an application. For example, in some cases, a user may scrollthrough an application, revealing different states that are viewedduring the scrolling. The analytics modules 212 may timestamp the eventsincluded in the logs. The timestamps may indicate when the eventsoccurred.

The UF determination system 108 may implement a function ontology thatmay be used to map function names to application states (e.g., mapfunction names to event analytics data to generate function analyticsdata as illustrated in FIG. 7). The UF determination system 108 may mapfunction names from the function ontology to one or more applicationsstates. For example the UF determination system 108 may map a singlefunction name to multiple states (e.g., thousands of states). Asdescribed herein, the mapping of function names to application statescan be used for analytics (e.g., to determine usage fraction values)and/or search.

Each application can be associated with a plurality of function names(e.g., a list of function names). The function names associated with anapplication may be descriptive of the different functionalities providedby the application. The UF determination system 108 can map functionnames associated with an application to application states of theapplication (e.g., using mapping instructions 706 of FIG. 7). Thefunction name mapped to a specific state may be descriptive of the stateto which the function name is mapped. For example, for a reviewapplication (e.g., business and restaurant reviews), the UFdetermination system 108 can map the function names “show restaurantreview” and “show business review” to individual states showingrestaurant and business reviews, respectively. As another example, for amovie database application (e.g., the internet movie database IMDb®application), the UF determination system 108 can map the function names“show actor information” and “show movie information” to individualstates that show actor information and individual states that show movieinformation, respectively. As another example, if an application allowsthe user to reserve restaurant tables, the list of functions associatedwith the application may include “Reserve Restaurant,” and the UFdetermination system 108 can map the function name “Reserve Restaurant”to different states of the application that provide reservationfunctionality for different restaurants. As described herein, the searchsystem 110 may assign function names to search records used to searchfor applications.

Although the UF determination system 108 may map a single function nameto multiple different states in an application, in some cases, the UFdetermination system 108 may map a single function name to only a singlestate instead of multiple states. For example, if a movie applicationincludes a single search page, the UF determination system 108 can mapthe function “Search Movies” to the single search page and may not mapthe function “Search Movies” to any other states within the application.

In some implementations, the UF determination system 108 and the searchsystem 110 may use a function ontology that is specific (e.g.,proprietary) to the systems 108, 110. For example, the search systemoperator and/or the UF determination system operator may implement aspecific function ontology that may or may not be used by other parties.In these cases, the UF determination system 108 can map functions tostates according to instructions developed by the UF determinationsystem operator and the search system operator (e.g., the same party insome cases). In some implementations, the UF determination system 108may use a function ontology provided by another party that assignsfunctions to states (e.g., an application developer, analytics provider,or other ontology source). In these cases, the UF determination system108 may map function names according to the other party's rules and thesearch system 108 may implement the techniques described hereinaccording to the ontology used by the UF determination system 108.

In some implementations, the UF determination system 108 may implement afunction ontology that is applicable across multiple applications suchthat different applications may use the same function names. In thisimplementation, the UF determination system 108 can apply the samefunction name to states of different applications. Such a functionontology applied uniformly across a plurality of applications may bereferred to as a generic function ontology. The function names of ageneric function ontology may be referred to as generic function names.The mapping of generic function names using a generic function ontologymay result in a uniform mapping of functions across differentapplications. This may be the case when application developers (oroperators of the UF determination system 108) map function names fromthe same function name schema. Uniform mapping of function names acrossdifferent applications may allow for comparison of states from differentapplications based on the function names assigned to the states. In someimplementations, function names used for an application may not beselected from a uniform function ontology that is applied across aplurality of applications. Instead, different applications may usedifferent function naming schemes. Function names used in this type ofscheme may be application-specific function names instead of genericfunction names. Application-specific function names may include theapplication name along with the function name, thereby indicating theapplication with which the function is associated.

FIG. 4C illustrates a state/function graph including states andfunctions from two different applications (A1 and A2). In FIG. 4C, thenotation indicating the application and state is A#-S#. For example, afirst state of a first application is labeled A1-S1. A first state of asecond application is labeled A2-S1. Note that the states associatedwith application A2 are shaded.

Functions are illustrated as broken lines around one or more states. Thegraph shows functions F1-F3 and F4-F5 for applications A1 and A2,respectively, although other functions may be included in applicationsA1 and A2. The states included with the same function may be assignedthe same function name. For example, states A1-S1, A1-S2, and A1-S4 areassigned the function name F1 (e.g., by the UF determination system108). States A1-S1, A1-S2, and A1-S4 may perform similar functions.

In some cases, a user can transition from a first state having a firstfunction name to a second state having a second function name that isdifferent than the first function name. For example, a transition fromstate A1-S2 to A1-S3 is a transition from function name F1 to functionname F2. In other cases, transitions between states can be within thesame function name (e.g., between state A1-S4 and state A1-S2).

The state/function graphs of FIGS. 4A-4C are for illustration purposesonly. For example, applications and functions may include a much largernumber of states than illustrated. In some cases, a single function mayinclude thousands or millions of different states. Accordingly, thefunctions and states included in the state/function graphs of FIGS.4A-4C are merely representations that an application can include one ormore functions and each function can include one or more states. Thestates available in an application and the functions associated with thestates may change over time. The UF determination system 108 and thesearch system 110 may update over time to reflect the changes inapplication states and functions.

The analytics data may initially be acquired by the user devices 106.Analytics data can be acquired by the user devices in a variety ofdifferent manners. In some implementations, a user device 106 caninclude one or more analytics modules 212 that acquire analytics data.For example, with respect to FIG. 2, native applications 210 can includeapplication analytics modules 212-1 that acquire analytics data. Asanother example, the user devices 106 may include other modules outsideof the installed native applications (e.g., a launcher application or anative application dedicated to analytics acquisition) that acquireanalytics data, which may be referred to as device analytics modules212-2. With respect to device analytics modules 212-2, the analyticsprovider systems 102 may provide device analytics modules 212-2 forinstallation on the user devices 106. For example, the analyticsprovider systems 102 may provide the device analytics modules 212-2 todevice manufacturers, launcher application developers, or other partiesthat may include the device analytics modules 212-2 on the user device106 outside of other native applications on the user device 106. In someexamples, a user device 106 may include a device analytics module 212-2and one or more application analytics modules 212-1 in one or moreapplications that acquire analytics data.

Analytics data may be categorized herein as event analytics data,function analytics data, and user engagement analytics data (hereinafter“user analytics data”). Event analytics data, function analytics data,and user analytics data are now described in turn.

Event analytics data refers to analytics data at the event level (e.g.,accessing an application state). For example, the event analytics datamay describe application usage in terms of application states accessedby users (e.g., in terms of access mechanisms or other deviceoperations). The event analytics data may indicate which states wereaccessed by users, the time at which the states were accessed by theusers (e.g., based on timestamps associated with an event), the amountof time spent by the users within the states, and users' selection ofGUI elements (e.g., user selectable links) within the states. Eventanalytics data may also include data indicating a user's path throughstates of one or more applications, such as a sequence of statesaccessed by the users. The events included in the events analytics datamay correspond to entry into (e.g., rendering of) an application stateor exit from an application state. The UF determination system 108(e.g., the function mapping module 504) may map function names to theevents.

Function analytics data refers to analytics data at the applicationfunction level. For example, the function analytics data may describeapplication usage in terms of functions accessed by users (e.g., interms of function names). The function analytics data may indicate whichfunctions were accessed by users (e.g., function names), the time atwhich the functions were accessed by the users (e.g., timestampsassociated with function names), the amount of time spent by the userswithin the functions, and users' selection of GUI elements within thefunctions. Function analytics data may also include data indicating auser's path through functions of one or more applications, such as asequence of functions accessed by the users. In some implementations,function analytics data may indicate when a user enters a functionand/or exists from a function. The UF determination system 108 candetermine usage fraction values based on the function analytics data.

In some implementations, the UF determination system 108 can receiveevent analytics data and generate function analytics data based on theevent analytics data. For example, the UF determination system 108 maymap function names to events (e.g., accessing an application state)included in the event analytics data in order to generate the functionanalytics data. The mappings of functions to events may be stored asfunction analytics data at the UF determination system 108. In general,the analytics data received at the UF determination system 108 mayinclude event analytics data that is then converted into functionanalytics data by the UF determination system 108. In some cases,however, the UF determination system 108 may receive function analyticsdata directly. For example, the UF determination system 108 may receivefunction analytics data in cases where the analytics modules 212generate function analytics data at the user devices 106. In otherexamples, the UF determination system 108 may receive function analyticsdata in cases where the application developer systems 100 and/or theanalytics provider systems 102 map function names to events in the eventanalytics data received from user devices 106.

User analytics data refers to analytics data indicating an amount ofuser engagement with an application (e.g., across a plurality of thestates/functions of the application). The user analytics data can begenerated by the analytics providers, application developers, and/or theUF determination system 108. In some implementations, user analyticsdata may include the number of users associated with the analytics datareceived by the UF determination system 108. For example, if eventanalytics data is collected for one thousand users, then the useranalytics data may indicate that one thousand users generated the eventanalytics data. As another example, the user analytics data may indicatethe number of user devices that generated the analytics data. In someimplementations, user analytics data may include a number of activeusers of the application (e.g., number of active monthly users), whichmay be provided by the application developer systems 100. In someimplementations, the analytics data may include other usage metrics,such as the number of times the application was downloaded, which may beprovided by the digital distribution platforms 104.

An application analytics module 212-1 may include code inserted into anative application 210 that causes the native application to acquireanalytics data. For example, the application analytics modules 212-1 mayacquire the event analytics data described herein. In someimplementations, the application analytics modules 212-1 may acquirefunction analytics data. The application analytics modules 212-1 mayalso acquire user analytics data (e.g., data used to determine userengagement values) indicating an amount of user engagement with theapplication.

After acquiring the analytics data, the application analytics module212-1 may store the analytics data on the user device 106 (e.g., inmemory of the user device 106) and/or transmit the analytics data forstorage and analysis on other systems. In some implementations, theapplication developer systems 100 may store the analytics data. Forexample, an application developer system 100 can have a storage systemfor storing the analytics data acquired from their applications. In someimplementations, an analytics provider system 102 can store theanalytics data acquired using their analytics modules. For example, ananalytics provider system 102 can store analytics data for a pluralityof native applications and provide the analytics data to applicationdeveloper systems upon request.

A device analytics module 212-2 may include a program that acquiresanalytics data similar to the application analytics modules 212-1 (e.g.,event analytics data, function analytics data, and/or user analyticsdata). After acquiring the analytics data, the device analytics module212-2 may store the analytics data on the user device 106 (e.g., inmemory of the user device 106) and/or transmit the analytics data forstorage and analysis on other systems. For example, the device analyticsmodule 212-2 may provide analytics data for a specific application tothe application developer system associated with the application. Asanother example, the analytics provider system can store the analyticsdata acquired using their analytics module and provide it to theapplication developer systems 100 and/or the UF determination system 108upon request.

In some implementations, analytics data may be collected by advertisers(not illustrated) that may then provide the analytics data to the UFdetermination system 108. For example, the advertisers may advertiselinks to application states and collect analytics indicating whichstates were accessed by users. Additional sources of analytics data maybe provided by search systems (e.g., search systems that provide accessto states of applications). For example, search systems may deliverlinks to states of applications and collect analytics data indicatingwhich links the users selected. Additional sources of analytics data maybe provided by operators of other computing systems that provide userselectable links to users in any manner, such as feeds (e.g., newsfeeds).

As described above, the analytics data collected by the analyticsmodules 212 can be stored on at least one of the user devices 106, theapplication developer systems 100, and the analytics provider systems102. The UF determination system 108 can retrieve the analytics datafrom the user devices 106, the application developer systems 100, and/orthe analytics provider systems 102. For example, the operators of the UFdetermination system 108 may enter into partnership arrangements whereapplication developers provide analytics data to the owners of thesearch system 110 and the UF determination system 108 in order toimprove relevance of search results associated with their applications.Although the search system 110 and UF determination system 108 can beoperated by the same party (e.g., the same business), in someimplementations, the search system 110 and UF determination system 108can be operated by separate parties. For example, one or more of theanalytics provider systems 102 may include the functionality of the UFdetermination system 108 described herein. In these implementations, thesearch system 110 may receive the function names and function metadatafrom the UF determination system 108 according to an agreement betweenthe search system operator and the UF determination system operator.

The UF determination system 108 can retrieve the analytics data inbatches. For example, the UF determination system 108 can retrievebatches of analytics data at preset times (e.g., at regular timeintervals). Additionally, or alternatively, the UF determination system108 can retrieve the analytics data when a predetermined amount ofanalytics data is available, such as when analytics data is availablefor a predetermined number of users, predetermined number of events,and/or predetermined number of functions. The batches of analytics datamay include data for a plurality of users that have accessed a pluralityof different states in different applications. In some cases, thebatches of data may include data from a single application (e.g.,analytics data for a plurality of users that accessed a plurality ofstates in the application). In some implementations, the UFdetermination system 108 can retrieve the analytics data in real time asthe data is collected for an application, instead of in batches.

The UF determination system 108 (e.g., the UF determination module 506)determines the usage fraction values for a plurality of applicationsbased on the analytics data (e.g., the event analytics data, functionanalytics data, and user analytics data). For example, the UFdetermination system 108 can determine event-based and/or time-basedusage fraction values based on the analytics data. The UF determinationsystem 108 may also determine user engagement values based on the useranalytics data. The search system 110 (e.g., the recordgeneration/update module 510) can update the search records (e.g.,search records 516) with function names and function metadata based onthe determined usage fraction values and the determined user engagementvalues. An example UF determination system 108 and search system 110 aredescribed with respect to FIG. 5.

Referring to FIG. 5, the UF determination system 108 includes ananalytics collection module 500, an analytics data store 502, a functionmapping module 504, and a usage fraction determination module 506(hereinafter “UF determination module 506”). The analytics collectionmodule 500 retrieves analytics data and stores the analytics data in theanalytics data store 502. The function mapping module 504 maps functionnames to the events (e.g., states) of the analytics data to generatefunction analytics data. The UF determination module 506 determinesusage fraction values based on the function analytics data. A userengagement determination module 508 (hereinafter “engagementdetermination module 508”) may determine user engagement values based onthe user analytics data.

The search system 110 includes a record generation/update module 510(hereinafter “record generation module 510”), a search data store 512,and an application search module 514 (hereinafter “search module 514”).The record generation module 510 retrieves function names and functionmetadata from the UF determination system 108 and updates search records516 stored in the search data store 512 to include the function namesand the function metadata. The search module 514 receives a querywrapper 518 (e.g., including a search query) from a querying user device520 and generates search results 522 based on the received query wrapper518 and the function names/metadata included in the search records 516.The search module 514 then transmits the search results 522 to thequerying user device 520.

The analytics collection module 500 retrieves analytics data from atleast one of the user devices 106, the analytics provider systems 102,and the application developer systems 100. In general, the analyticsdata received by the analytics collection module 500 may include anydata that can be used by the UF determination system 108 to determinethe various usage fraction values described herein. The analyticscollection module 500 may store the analytics data in analytics records524 in the analytics data store 502. The analytics data retrieved by theanalytics collection module 500 may include event analytics data,function analytics data, and/or user analytics data. In some cases, theanalytics collection module 500 may retrieve event analytics data andthe function mapping module 504 may generate function analytics databased on the event analytics data. In other cases, the analyticscollection module 500 may receive function analytics data directly fromthe user devices 106, the analytics provider systems 102, and/or theapplication developer systems 100.

FIG. 6 illustrates an example analytics record 524. The analytics record524 may represent analytics data and various values associated with asingle application. The analytics data store 502 stores a plurality ofsuch analytics records 524, each of which may correspond to a differentapplication. The analytics data store 502 (e.g., the analytics records524) may be updated and/or accessed by modules of the UF determinationsystem 108 and the search system 110. The records described herein(e.g., the analytics records 524, function mapping records 526, andsearch records 516) represent data stored in the data stores (e.g., theanalytics data store 502, function data store 528, and search data store512). The data stores may include a variety of different data structuresthat are used to implement the techniques of the present disclosure,such as one or more databases, indices (e.g., inverted indices), tables,files, or other data structures.

The analytics record 524 includes an analytics record name/ID 600(hereinafter “analytics record ID 600”) that identifies the analyticsrecord 524 (e.g., uniquely identifies) among the other analytics recordsin the analytics data store 502. In implementations where the analyticsdata store 502 includes a single analytics record 524 for a singleapplication, the analytics record ID 600 may identify the applicationwith which the analytics data and values of the analytics record 524 isassociated. In some cases, the analytics record ID 600 may include theapplication name and application edition with which the analytics record524 is associated. For example, the analytics record ID 600 may includean application ID that uniquely identifies the application edition withwhich the analytics record 524 is associated. As described hereinafter,it can be assumed that an analytics record 524 represents analytics dataand values for a single application.

The analytics record 524 includes analytics data 602 for theapplication. For example, the analytics record 524 may include eventanalytics data 602-1, function analytics data 602-2, and user analyticsdata 602-3. The analytics data 602 included in the analytics record 524may be retrieved from a variety of different sources. The analyticsrecord 524 may also include usage fraction values 604 and userengagement values 606 for the application. Retrieval and determinationof the analytics data, usage fraction values, and engagement values isdescribed herein.

Referring to FIG. 5, the function mapping module 504 maps function namesto events (e.g., accessed application states). As described herein, theanalytics data including function names (e.g., mapped to events) may bereferred to as function analytics data. Accordingly, an example sourceof function analytics data may be the mapping of function names to eventanalytics data, as provided by the function mapping module 504. In othercases, function analytics data may be provided by at least one of theuser devices 106, the analytics provider systems 102, and theapplication developer systems 100. The function mapping module 504 mayretrieve event analytics data from the analytics data store 502 (e.g.,analytics records 524) and map function names to events in the eventsanalytics data. The function mapping module 504 may map function namesto events based on data included in function mapping records 526 storedin the function data store 528.

FIG. 7 illustrates an example function mapping record 526. A functionmapping record 526 may include data indicating how to map function namesto events (e.g., an opening of an application state) for an application(e.g., a single native application). The function data store 528 mayinclude a plurality of function mapping records similar to the functionmapping record 526 of FIG. 7, each of which may include data indicatinghow to map function names to events for a respective application.

The function mapping record 526 may include a mapping record ID 700 thatidentifies the mapping record 526 (e.g., uniquely identifies) among theother mapping records in the function data store 528. The functionmapping record 526 may also include an application mapping name/ID(hereinafter “app mapping ID 702”). The app mapping ID 702 may indicatethe application with which the function mapping record 526 isassociated. For example, the app mapping ID 702 may indicate theapplication with which the function name list 704 and the functionmapping instructions 706 are associated. In some implementations, theapp mapping ID 702 may correspond to an application named in theanalytics record name/ID 600 and an application name/ID in a searchrecord 516. For example, if an application that provides restaurantreviews is named “ReviewsApp,” then the analytics record and thefunction mapping record for the “ReviewsApp” may include an analyticsrecord ID and an app mapping ID that reference the “ReviewsApp” (e.g.,include an identifier for the ReviewsApp and/or the name “ReviewsApp”).Additionally, the search data store 512 may include a search record 516for the “ReviewsApp” application.

The function name list 704 includes list of function names associatedwith the application referenced by the app mapping ID 702. The list offunctions may include those functions that can be mapped to eventanalytics data by the function mapping module 504. For example, a reviewapplication that provides reviews of businesses (e.g., the YELP®application developed by Yelp, Inc.) may have a list of function namesthat include “Read Restaurant Review,” “Read Store Review,” “ProvideRestaurant Info,” “Show menu,” and “Show food photos.” As anotherexample, a mapping and navigation application may have a list offunction names that include “Find transportation to location,” “DrivingDirections,” “Walking Directions,” and “Show nearby gas stations.”

The function mapping record 526 includes function mapping instructions706 (hereinafter “mapping instructions 706”). The mapping instructions706 (e.g., rules) indicate how the function names of the function namelist 704 can be mapped to events included in the event analytics data.The function mapping module 504 maps the function names of the functionname list 704 to events in the event analytics data based on the mappinginstructions 706. For example, the function mapping module 504 mayretrieve an event for an application from event analytics data, retrievethe function mapping record for the application, and then map (e.g.,assign) a function name to the event based on mapping instructions thatindicate which function name to map to the event. The function mappingmodule 504 may store the mapping of the function name to the event inthe analytics data store 502 as function analytics data. In this manner,the function mapping module 504 may generate function analytics databased on event analytics data. For example, the function mapping module504 may perform mappings for batches of events (e.g., for single ormultiple applications). As another example, the function mapping module504 may perform mappings for events as the events are received by theanalytics collection module 500 (e.g., in real time).

As described above, events may indicate accessed application/web states(e.g., access mechanisms such as URLs) that correspond to functions,application/OS operations that are indicative of functions (e.g., anAndroid intent/activity that launches a screen of an application), userinputs that are indicative of a state/function (e.g., selection of GUIinput elements), and/or other types of events, such as analytics modulespecific events (e.g., specific events coded by the analyticsdevelopers). The mapping instructions 706 may vary depending on whatinformation is included in the event analytics data. In someimplementations, the mapping instructions 706 may include instructionsfor mapping function names to application/web states (e.g., accessmechanisms) if application/web states are indicated in the eventanalytics data. For example, the mapping instructions 706 may instructthe function mapping module 504 to map function names based on thedomain of the access mechanism and the path of the access mechanism. Insome implementations, the mapping instructions 706 may includeinstructions for mapping function names to application/OS operations ifapplication operations are indicated in the event analytics data. Forexample, the mapping instructions 706 may instruct the function mappingmodule 504 to map function names to specific operations, such asspecific Android intents/activities. In some implementations, themapping instructions 706 may include instructions for mapping functionnames to user inputs, such as the selection of GUI elements havingspecific functionality (e.g., a search button or a category selectionbutton), if the selection of GUI elements is indicated in the eventsanalytics data. The types of events included in the event analytics datamay also vary based on the level of access the analytics module 212 hasto acquire analytics data. For example, an application analytics module212-1 included in a native application may have a greater level ofaccess to determine particular application states and operations of thenative application than a device analytics module 212-2 that is notassociated with (e.g., coded into) the application.

In some cases, the analytics data received by the analytics collectionmodule 500 includes function analytics data (e.g., analytics data withmapped function names). In these cases, the function analytics data mayinclude mapped function names upon receipt at the analytics collectionmodule 500 and may be stored in the analytics records 524 without anyadditional mapping operations being performed by the function mappingmodule 504.

FIG. 7 illustrates the function mapping module 504 mapping functionnames to various events. Specifically, the function mapping module 504retrieves three events from event analytics data. The retrieved eventsare for an application named “App 1.” Each of the events includes atimestamp, labeled as Time 1, Time 2, and Time 3. The timestampsindicate the time at which the events occurred. The function mappingmodule 504 retrieves the function mapping record for App 1 and mapsfunction names from the function name list to the events according tothe function mapping instructions for App 1. In FIG. 7, Function name 1maps to Event 1 and Event 3. Function name 2 maps to Event 2. Thefunction mapping module 504 may generate function analytics data for aplurality of events for each of a plurality of applications as describedabove. In terms of the graph of FIG. 4C, the sequence of 3 events may bethe entry into state A1-S2 at Time 1 followed by entry into state A1-S3at Time 2 and the return to either state A1-S2 or transition to A1-S1 atTime 3. Such a path through the graph of FIG. 4C would transition fromA1-F1 to A1-F2 and back to A1-F1.

The UF determination module 506 may determine the amount of time thatwas spent in an event (e.g., in a state/function) based on thetimestamps for single events indicating when the events occurred. Forexample, assuming that Event 1, Event 2, and Event 3 occur in sequence,the timestamps Time 1, Time 2, and Time 3 may be sequential. Thedifference between Time 3 and Time 2 may indicate an amount of time theuser spent in Function Name 2. Accordingly, the UF determination module506 can determine the amount of time spent in Function Name 2 based onthe difference between Time 3 and Time 2. Although the UF determinationmodule 506 may determine the amount of time spent in an event/functionin some cases, in other cases, the event/function analytics datareceived by the analytics collection module 500 may already indicate theamount time spent in the event/function.

The function mapping record 526 includes function metadata 708. Thefunction metadata 708 may include metadata associated with each of thefunction names in the function name list. The metadata associated with afunction name may include various terms (e.g., words and/or phrases)that are associated with the function name and may provide context as tothe meaning of the function name. For example, if the function name is“Navigate to a location,” the metadata associated with the function namemay include terms related to navigating to a location, such as“transit,” “directions,” “map,” “car,” “on foot,” and “by bike.” Asanother example, if the function name is “Order food,” the metadataassociated with the function name may include terms related to orderingfood, such as “takeout,” “pickup,” “order online,” “delivery,” and“local restaurant.” The terms in the function metadata 708 may be wordsthat are commonly associated with the functions. In some examples, thefunction metadata 708 may include terms that are commonly used to searchfor applications that provide the functions. For example, the functionmetadata 708, along with the function name, may include terms indicatingthe functions a user wants to perform with the application. As such, thefunction metadata 708 may include a group of terms that may enhanceapplication search relevance when assigned to the search records 516.The function metadata can be generated manually and/or automatically.

Referring to FIG. 5, the UF determination module 506 determines usagefraction values based on the function analytics data. For example, theUF determination module 506 may retrieve function analytics data for asingle application and determine one or more types of usage fractionvalues for the application. The UF determination module 506 may storethe usage fraction values in the analytics record 524 for theapplication.

As described herein, each application has a set of associated functionnames. The UF determination module 506 can determine a usage fractionvalue for each of the function names associated with the application.The usage fraction value may indicate an amount of usage of the functionwith which the usage fraction value is associated. Put another way, ausage fraction value may indicate the amount of interaction a user haswith an application function relative to other functions of theapplication. The usage fraction value may be expressed as a percentagevalue indicating how much the function is used relative to otherfunctions of the application. The usage fraction values for a list offunctions may add up to one (e.g., 100%). For example, for anapplication that has three functions (i.e., function names), there maybe three usage fraction values (i.e., one for each function). In thisexample, the three usage fraction values may add up to one.

The UF determination module 506 may calculate different types of usagefraction values (e.g., a time-based usage fraction value and/or anevent-based usage fraction value). The different types of usage fractionvalues may reflect the different ways in which application usage can bequantified. One type of usage fraction value described herein may bereferred to as a time-based usage fraction value. The time-based usagefraction value may indicate the amount of time users spent accessing onefunction of an application relative to other functions of theapplication. For example, a time-based usage fraction value may indicatehow much time users spent in states associated with a particularfunction relative to the time the users spent in states associated withother functions of the application.

Another type of usage fraction value described herein may be referred toas an event-based usage fraction value. An event-based usage fractionvalue may indicate how users access states associated with anapplication's functions and navigate through states associated with thefunctions. An example event-based usage fraction value may indicate howmany times states of a first function are accessed relative to states ofother functions. For example, the UF determination module 506 maycalculate a total number of times functions (e.g., states) of theapplication were accessed and then determine an event-based usagefraction value for each function by determining the fraction of totalaccesses that are attributable to each function. The UF determinationmodule 506 can determine a variety of different types of event-basedusage fractions.

In some cases, a set of function names can be associated with more thanone type of usage fraction value in the analytics record 524. Forexample, if the UF determination module 506 calculates two differentusage fraction values for an application, each function name for theapplication may have two assigned usage fraction values. Morespecifically, if an application includes three functions, each of thethree functions may be assigned two usage fraction values. The sum ofusage fraction values for each type of usage fraction may be one (e.g.,100%). In an example where an application includes three functions andtwo types of usage fraction values, the sum of the usage fraction valuesfor the first type may be one. Additionally, the sum of the usagefraction values for the second type of usage fraction value may be one.FIG. 6 illustrates two different examples of usage fraction valuesassigned to function names. In a first example at 608, the applicationis associated with four function names. Each of the function names isassociated with one usage fraction value. The four usage fraction valuesat 608 may add up to one. In a second example at 610, the application isassociated with two different types of usage fraction values. Usagefraction values UF1-UF4 are a first type of usage fraction value. Usagefraction values UF1-UF4 may add up to one. Usage fraction values UF5-UF8are a second type of usage fraction value that is different than thefirst usage fraction value. Usage fraction values UF5-UF8 may add up toone.

FIGS. 8-9 illustrate methods for determining usage fraction values. FIG.8 illustrates an example method for generating time-based usage fractionvalues for an application. The method of FIG. 8 is now described withreference to the UF determination system of FIG. 5. At block 800, theanalytics collection module 500 retrieves function analytics data for anapplication. At block 802, the UF determination module 506 determines atotal amount of time spent by users across the set of functions for theapplication. At block 804, the UF determination module 506 determinesthe amount of time spent by users within each of the functions for theapplication. At block 806, for each function of the application, the UFdetermination module 506 determines a time-based usage fraction value bydividing the time spent in the function by the total amount of timespent by users across the set of functions for the application. In block808, the UF determination module 506 stores the time-based usagefraction values in the analytics data store 502. The UF determinationmodule 506 can perform the method of FIG. 8 for a plurality ofapplications to generate time-based usage fraction values for theplurality of applications.

FIG. 9 illustrates an example method for generating event-based usagefraction values for an application. The method of FIG. 9 is nowdescribed with reference to the UF determination system of FIG. 5. Atblock 900, the analytics collection module 500 retrieves functionanalytics data for an application. At block 902, the UF determinationmodule 506 determines a total number of times users accessed the set offunctions for the application. At block 904, the UF determination module506 determines the number of times each of the functions was accessed.At block 906, for each function of the application, the UF determinationmodule 506 determines an event-based usage fraction value by dividingthe number of times the function was accessed by the total number oftimes users accessed the set of functions for the application. At block908, the UF determination module 506 stores the event-based usagefraction values in the analytics data store 502. The UF determinationmodule 506 can perform the method of FIG. 9 for a plurality ofapplications to generate event-based usage fraction values for theplurality of applications.

Although the method of FIG. 9 describes calculating an event-based usagefraction value based on the total number of times functions wereaccessed, in other examples, other types of event-based usage fractionscan be determined. For example, in some implementations, the event-basedusage fraction values can be based on the total number of times usersentered into functions, as opposed to the number of times the functionswere accessed. More specifically, if a user first enters a function andthen transitions to other states of the same function, such a series oftransitions may be counted as a single event for the purposes ofcalculating an event-based usage fraction value. This is in contrast tothe calculation of FIG. 9 where the UF determination module 506 maycount each accessed function as an event for the purposes of calculatingthe event-based usage fraction values.

Although usage fraction values may be determined based on analytics datacollected for a plurality of different users, in some implementations,the UF determination system 108 may determine usage fraction values fora single user. For example, the UF determination module 506 maydetermine time-based usage fraction values and/or event-based usagefraction values for an individual user based on event analytics data andfunction analytics data collected for the individual user.

The engagement determination module 508 determines one or more userengagement values for each application. The user engagement valuesindicate an amount of users that have engaged with an application. Theengagement determination module 508 may determine a user engagementvalue that indicates the number of users for which analytics data wascollected for the application. For example, the engagement determinationmodule 508 may determine the user engagement value based on the numberof users associated with the analytics data used to determine the usagefraction values. Additionally, or alternatively, the engagementdetermination module 508 may determine user engagement values usingother data. In some implementations, the engagement determination module508 may determine a user engagement value based on the number of timesthe application was downloaded from a digital distribution platform 104.In some implementations, the engagement determination module 508 maydetermine a user engagement value based on the number of active users ofthe application (e.g., the number of daily active users and/or monthlyactive users). The engagement determination module may store one or moreuser engagement values in the analytics data store 502.

FIG. 10 illustrates an example method describing operation of the UFdetermination system 108 with respect to a plurality of applications. At1000, the analytics collection module 500 receives analytics data for aplurality of applications over time and may store the analytics data inthe analytics data store 502. At 1002, the function mapping module 504retrieves event analytics data and maps (e.g., assigns) function namesto events in the event analytics data to generate function analyticsdata. At 1004, the UF determination module 506 determines usage fractionvalues for the plurality of applications based on the function analyticsdata. At 1006, the engagement determination module 508 may optionallydetermine one or more user engagement values for the plurality ofapplications.

Referring to FIG. 5, the search system 110 receives query wrappers 518(e.g., search queries) from user devices 520 (which may be referred toas “querying user devices 520”), generates search results 522 inresponse to the query wrappers 518, and transmits the search results tothe user devices 520. The search system 110 may use function names,function metadata, usage fraction values, and/or user engagement valuesin order to generate the search results 522. The user device 520 mayrender the search results 522 as user selectable links on a searchengine results page (SERP) (e.g., FIG. 11) which the user may select inorder to preview/download native applications included in the searchresults 522.

The search module 514 receives the query wrapper 518 (e.g., including asearch query) from a user device 520 and generates search results 522based on data included in the search data store 512. The search datastore 512 includes search records 516 (e.g., search records 516-1, 516-2of FIGS. 12A-12B) that include data used during search and forgenerating search results 522. For example, the search records 516 caninclude data (e.g., function names and/or function metadata) related toapplications that may be downloaded from the digital distributionplatforms 104.

The record generation module 510 generates the search records 516. Therecord generation module 510 may retrieve data from data sources andgenerate the search records 516. The record generation module 510 mayalso retrieve data from the UF determination system 108 and generateportions of the search records 516 to enhance the data included in thesearch records 516 with data generated by the UF determination system108. For example, the record generation module 510 may insert functionnames, function metadata, one or more usage fraction values, and/or oneor more user engagement values in the search records 516.

Search system operation is described hereinafter in detail with respectto FIGS. 11-15. FIG. 11 illustrates an environment including a queryinguser device 520 and a search system 110 that generates search results522 for the user device 520. FIGS. 12A-12B illustrate example searchrecords 516-1, 516-2 stored in the search data store 512. FIG. 13illustrates a method for assigning function names and/or functionmetadata to search records. FIGS. 14-15 describe operation of an examplesearch module 514.

FIG. 11 illustrates an example user device 520 that transmits a querywrapper 518 to the search system 110 via the network 112 and receivessearch results 522 from the search system 110. As described herein,analytics data may be retrieved from a plurality of user devices 106.Some of the plurality of user devices 106 may optionally interact withthe search system 110. A user device 520 that interacts with the searchsystem 110 to retrieve search results 522 may be referred to herein as a“querying user device 520.” In some cases, querying user devices 520 mayprovide analytics data via analytics modules 212. In other cases,querying user devices 520 may not include the analytics modules 212and/or provide analytics data. For example, the UF determination system108 may generate the usage fraction values, user engagement values,function names, and function metadata based on interaction with a firstset of user devices 106 that provide analytics data and the searchsystem 110 may provide results to a plurality of querying user devices520 not included within the first set of user devices 106 that providedthe analytics data.

In some examples, user devices 520 may communicate with the searchsystem 110 via a partner computing system (not illustrated). The partnercomputing system may be a computing system of a third party that mayleverage the search functionality of the search system 110. The partnercomputing system may belong to a company or organization other than thatwhich operates the search system 110. Example third parties that mayleverage the functionality of the search system 110 may include, but arenot limited to, internet search providers and wireless communicationsservice providers. The user devices 520 may send search queries to thesearch system 110 and receive search results 522 via the partnercomputing system. The partner computing system may provide a userinterface to the user devices 520 in some examples and/or modify thesearch experience provided on the user devices 520.

In general, a user device 520 may communicate with the search system 110using any application that can transmit search queries to the searchsystem 110. In some examples, a user device 520 may execute a nativeapplication that is dedicated to interfacing with the search system 110,such as a native application dedicated to searches (e.g., searchapplication 1100). In some examples, a user device 520 may communicatewith the search system 110 using a more general application, such as aweb-based application accessed using the web browser application 1102(e.g., a web search). In still other examples, the functionalityattributed to the search application 1100 herein may be included as asearching component of a larger application that has additionalfunctionality. For example, the functionality attributed to the searchapplication 1100 may be included as part of a native/web application asa feature that provides search for the native/web application. Althoughthe user device 520 may communicate with the search system 110 using aweb based application and/or a native search application, the userdevice 520 may be described hereinafter as using the native searchapplication 1100 to communicate with the search system 110.

The search application 1100 may display a search field on a graphicaluser interface (GUI) in which the user can enter search queries 1104.The user may enter a search query 1104 into the search field (e.g., asearch box) using a touchscreen or physical keyboard, a speech-to-textprogram, or other form of user input. In general, a search query 1104may be a request for information retrieval (e.g., search results 522)from the search system 110. For example, a search query may 1104 bedirected to retrieving a list of links to preview/download applicationsfrom the search system 110. A search query 1104 may include text,numbers, and/or symbols (e.g., punctuation) entered into the user device520 by the user. A user may enter a variety of different types of searchqueries. In some cases, a user may enter a search query indicating whatthey want to do with an application. For example, a user may enter termsin the search query 1104 that describe functions the user wants anapplication to perform (e.g., navigate, provide business reviews, etc.)and/or terms associated with the functions that the user wants anapplication to perform (e.g., “directions,” “map,” “car,” “on foot,” and“by bike”). Including function names and/or function metadata in thesearch records 516 may increase the likelihood that search records arematched to such queries.

The user device transmits a query wrapper 518 to the search system 110that may include a search query 1104. The query wrapper 518 may includeadditional data along with the search query 1104. For example, the querywrapper 518 may include geo-location data 1108 that indicates thelocation of the user device 520 (e.g., latitude/longitude coordinates).The query wrapper 518 may also include additional data, including, butnot limited to, platform data 1110 (e.g., version of the operatingsystem 1112, device type, and web-browser version), an identity of auser of the user device 520 (e.g., a username), partner specific data,ISP/hostname, and other data.

A user device 520 receives a set of search results 522 from the searchmodule 514 that are responsive to the query wrapper 518. The searchresults 522 received from the search module 514 may include one or moreapplication download addresses 1106, result scores 1114, and link data1116. The user device 520 (e.g., the search application 1100) rendersuser selectable links 1118 based on the received search results 522(e.g., links 1118-1, 1118-2, . . . , 1118-5 of FIG. 11). For example,the user device 520 may render user selectable links based on the linkdata 1116. The link data 1116 may include data for rendering the userselectable links, such as text and/or image data to be included in theuser selectable link 1118. Example link data 1116 may include text andimages including, but not limited to, application names associated withthe application download addresses 1106, text describing theapplication, and images associated with the application (e.g.,application icons).

Each user selectable link 1118 displayed to the user may include one ormore application download addresses 1106 (e.g., for different digitaldistribution platforms 104 and/or operating systems) and be associatedwith a result score 1114 that indicates the relevance of the userselectable link 1118 (e.g., the application to be downloaded) to thesearch query 1104. A user may select a user selectable link 1118 byinteracting with the link (e.g., touching or clicking the link). Inresponse to selection of a link, the user device 520 may access thedigital distribution platform 104 referenced by the application downloadaddress 1106 associated with the user selectable link 1118. In someimplementations, the application download address 1106 may initiate adownload of the application. In other implementations, the applicationdownload address may access a site where the application can bedownloaded. For example, the site may provide a preview of theapplication (e.g., screenshots), a description of the application,ratings, a number of downloads, user reviews, and a link for downloadingthe application. In some cases, the site may be operated by the digitaldistribution platform 104 that provides the application for download. Inother cases, the site may be operated by the search system 110, whichmay provide application search services for a plurality of digitaldistribution platforms 104.

FIG. 11 illustrates an example list of user selectable links 1118 that auser device 520 may display to a user. Specifically, FIG. 11 includesuser selectable links 1118 included in search results for the searchquery “Late night food.” The user selectable links 1118 include links tonative applications including the YELP® native application, the DOMINOSPIZZA® native application developed by Domino's Pizza LLC, theTRIPADVISOR® native application developed by TripAdvisor, Inc., theOPENTABLE® restaurant reservation native application, and the GRUBHUB®Food Delivery native application developed by GrubHub, Inc. The userselectable links 1118 are rendered based on link data 1116 includingapplication icons and application names indicating the application thatwill be previewed/downloaded in response to selection of the links 1118.

The search data store 512 includes a plurality of search records 516.The search module 514 may use data included in search records 516 andthe query wrapper 518 in order to generate the search results 522. Forexample, the search module 514 may identify a plurality of searchrecords 516 based on the data included in the query wrapper 518 and thenfilter/score the identified search records 516 based on the dataincluded in the search records 516 and the query wrapper 518. The searchmodule 514 may then generate search results 522 based on the scoredsearch records 516.

FIGS. 12A-12B illustrate example search records 516-1, 516-2 that may beincluded in the search data store 512. Each search record 516 mayinclude data related to an application. The record generation module 510may generate and update the search records 516 based on data retrievedfrom a plurality of data sources (not illustrated). For example, therecord generation module 510 may automatically create new search records516 and update existing search records 516 based on data retrieved fromthe data sources. The record generation module 510 can extract and/orinfer the data included in the search records 516 (e.g., applicationattributes 1204) from documents retrieved from the data sources.Additionally, or alternatively, the data included in the search records516 (e.g., the application attributes 1204) may be manually generateddata. The record generation module 510 may update the search records(e.g., application attributes 1204) over time so that up-to-date searchresults can be provided in response to a search query.

The data sources may include a variety of different data providers andthe types of data retrieved from the data sources can include a varietyof different types of data related to applications. The data sources mayinclude data from application developers, such as applicationdevelopers' websites and data feeds provided by developers. The datasources may include operators of digital distribution platforms 104configured to distribute native applications to user devices 520. Thedata sources may also include other websites, such as websites thatinclude web logs (i.e., blogs), application review websites, or otherwebsites including data related to applications. Additionally, the datasources may include social networking sites, such as “FACEBOOK®” byFacebook, Inc. (e.g., Facebook posts) and “TWITTER®” by Twitter Inc.(e.g., text from tweets). Data sources may also include online databasesthat include, but are not limited to, data related to movies, televisionprograms, music, and restaurants. Data sources may also includeadditional types of data sources in addition to the data sourcesdescribed above. Different data sources may have their own content andupdate rate.

Examples of data related to applications include, but are not limitedto, a name of an application, a description of an application, asubstantive review of an application, a quality rating of anapplication, a developer name, an excerpt from a blog post about anapplication, a tweet about an application, and one or more images (e.g.,icons and/or screenshots) associated with the application. The recordgeneration module 510 may also retrieve statistical data from the datasources. Statistical data may include any numerical data related to anapplication, such as a number of downloads, download rates (e.g.,downloads per month), a number of reviews, and a number of ratings. Insome examples, data retrieved from the data sources may includeinformation regarding the functionalities of applications.

FIGS. 12A-12B illustrate example search records 516. Referring now toFIG. 12A, an example search record 516-1 includes an application name1200, an application identifier (ID) 1202, application attributes 1204,one or more application download addresses 1206, and link data 1208. Thesearch record 516-1 represents data stored in the search data store 512that is related to an application. The search data store 512 may includea plurality of search records having a similar structure as the searchrecord 516-1. Put another way, the search data store 512 may include aplurality of search records having an application name 1200, anapplication ID 1202, application attributes 1204, one or moreapplication download addresses 1206, and link data 1208.

The application name 1200 may be the name of the application representedby the data in the search record 516-1. Example application names mayinclude, Yelp, OpenTable, Google Maps, Facebook, Twitter, MicrosoftWord, or Angry Birds. The application ID 1202 identifies the searchrecord 516-1 among the other search records included in the search datastore 512. For example, the application ID 1202 may uniquely identifythe search record 516-1. The application ID 1202 may be a string ofalphabetic, numeric, and/or symbolic characters (e.g., punctuationmarks).

The one or more application download addresses 1208 may indicate asite(s) (e.g., web/native, such as the GOOGLE PLAY® digital distributionplatform) where the native application referenced in the search record516-1 can be downloaded and/or previewed. Link data 1208 may includedata used by the user device 520 to generate the user selectable links1118 (e.g., text and/or images) in the SERP (e.g., FIG. 11).

The application attributes 1204 may include any type of data which maybe associated with the application represented by the search record516-1. The application attributes 1204 may include a variety ofdifferent types of data. For example, the application attributes 1204may include structured, semi-structured, and/or unstructured data. Theapplication attributes 1204 may include information that is extracted orinferred from documents retrieved from the data sources. In someexamples, the application attributes 1204 may include data that ismanually generated. The application attributes 1204 may be updated sothat up to date results can be provided in response to a search query1104.

The application attributes 1204 may include the name of the developer ofthe application, the publisher of the application, a category (e.g.,genre) of the application, a description of the application (e.g., adeveloper's description), a version of the application, the operatingsystem for the application, and the price of the application. Theapplication attributes 1204 may also indicate security or privacy dataabout the application, battery usage of the application, and bandwidthusage of the application.

The application attributes 1204 may include application statistics.Application statistics may refer to numerical data related to theapplication. For example, application statistics may include, but arenot limited to, a number of downloads, a download rate (e.g., downloadsper month), a number of ratings, and a number of reviews. Theapplication attributes 1204 may also include information retrieved fromwebsites such as reviews associated with the application, articlesassociated with the application (e.g., wiki articles), or otherinformation. The application attributes 1204 may also include digitalmedia related to the application, such as images (e.g., icons and/orscreenshots).

The application attributes 1204 may also include the function names,function metadata, usage fraction values, and/or user engagement valuesdescribed herein. The record generation module 510 may retrieve thefunction names, function metadata, usage fraction values, and/or userengagement values from the UF determination system 108 (e.g. fromanalytics records 524 and/or function mapping records 526). For example,the record generation module 510 may retrieve the data from theanalytics records 524 by identifying matches between the applicationname 1200 and/or the application ID 1202 of the search record 516-1 andthe analytics record name/ID 600 of the analytics records 524. Asanother example, the record generation module 510 may retrieve the datafrom the function mapping records 526 by identifying matches between theapplication name 1200 and/or the application ID 1202 of the searchrecord 516-1 and the app mapping ID 702 of the function mapping record526.

In some cases, the record generation module 510 may be configured toassign each of the function names, associated function metadata, andassociated usage fraction values to the search record 516. In othercases, the record generation module 510 may be configured to assign asubset of the function names, associated function metadata, andassociated usage fraction values to the search record 516-1 (e.g., basedon assignment conditions described herein). In some cases, some searchrecords 516 may not include function names and function metadata.

FIG. 12B illustrates an example search record 516-2 for an applicationcalled “MappingApp.” The search record 516-2 includes the applicationname “MappingApp” 1202 and an application ID “ID#” 1202. The searchrecord 516-2 also includes link data 1208 for generating a userselectable link 1118 for the MappingApp and one or more applicationdownload addresses 1206 for downloading the MappingApp. The searchrecord 516-2 also includes application attributes 1204. The applicationattributes 1204 include the developer of the MappingApp, a genre of theMappingApp (e.g., travel), a description of the MappingApp (e.g., adeveloper's provided description), reviews of the MappingApp (e.g., userreviews), ratings (e.g., user ratings), and a number of times theMappingApp was downloaded.

The application attributes 1204 includes a single function name 1204-1associated with the MappingApp along with associated function metadata1204-2 and one or more usage fraction values 1204-3. The applicationattributes 1204 also includes one or more user engagement values 1204-4associated with the MappingApp. An example function name may be“Navigate to a location.” Example function metadata associated with the“Navigate to a location” function name may be “directions,” “map,”“car,” “on foot,” and “by bike.” The usage fraction values 1204-3 may bevalues between 0-100%, depending on how much the function “Navigate to alocation” is accessed by users of the MappingApp. Other example functionnames that may be associated with the MappingApp, but were not includedin the search record 516-2 (e.g., due to low usage fraction values) mayinclude “show traffic,” “show terrain,” and “show public transit,” forexample.

As described herein, the record generation module 510 may communicatewith the UF determination system 108 in order to retrieve functionnames, function metadata, usage fraction values, and user engagementvalues for assignment to search records 516. The function names and/orfunction metadata may be generally referred to hereinafter as “functionkeywords.” The function keywords, associated usage fraction values, andassociated user engagement values may be used in identification andscoring of search records 516 during search.

The record generation module 510 may assign function keywords to thesearch records 516 based on the satisfaction of function keywordassignment conditions (hereinafter “assignment conditions”) associatedwith the function keywords. In some implementations, the search systemoperator may set the assignment conditions. A variety of assignmentconditions are described hereinafter. In some implementations, thesearch system operator may use only a single assignment condition. Inother cases, the search system operator may use multiple assignmentconditions from the assignment conditions described hereinafter. Theassignment conditions may be stored in the record generation module 510and applied across a plurality of the search records 516. In otherimplementations, a search record may include app-specific assignmentconditions (not illustrated in FIGS. 12A-12B) indicating conditions forassigning function keywords to the search record.

Assignment conditions may vary, depending on how the search systemoperator implements the assignment conditions. In some implementations,the assignment conditions may include conditions regarding the magnitudeof usage fraction values associated with functions. In one example, anassignment condition may include a threshold usage fraction value. Forexample, the record generation module 510 may be configured to assignfunction keywords to the search record 516 if the usage fraction valueassociated with the function keywords (e.g., the function name) isgreater than a threshold value (e.g., >50%). As another example, therecord generation module 510 may be configured to assign functionkeywords to the search record 516 if the usage fraction value associatedwith the function keywords indicates that the function is used athreshold amount more than the other functions of the application (e.g.,the usage fraction value is a threshold amount greater than the otherusage fraction values). A function that is used a threshold amount morethan other functions may be referred to as a “dominant function” or a“specialized function” of the application, as the users may tend toassociate the application with that particular functionality. In someimplementations, the record generation module 510 may be configured toassign function keywords to the search record 516 if multiple usagefraction values of different types are greater than a threshold value.For example, the record generation module 510 may assign functionkeywords to a search record if an event-based usage fraction value and atime-based usage fraction value associated with the function keywordsare both greater than a threshold value. In some cases, the differenttypes of usage fraction values may be compared against the samethreshold value. In other cases, each type of usage fraction value maybe compared against a different threshold value.

In some implementations, the assignment conditions may includeconditions associated with a combination of a usage fraction value anduser engagement value. For example, the record generation module 510 maybe configured to assign function keywords to a search record 516 if theproduct of the usage fraction value and the user engagement value(referred to herein as a “normalized usage fraction value”) is greaterthan a threshold value. In a more specific example, if the usagefraction value (e.g., time or event-based usage fraction value) is 25%and the user engagement value (e.g., total monthly active users) is20000, then the normalized usage fraction value may equal 5000. If theassignment conditions include a threshold normalized usage fractionvalue of 4000, then the record generation module 510 may assign thefunction keywords associated with the normalized usage fraction value tothe search record 516.

In some cases, the assignment conditions may include a threshold userengagement value. For example, the record generation module 510 may beconfigured to assign function keywords to a search record 516 if theuser engagement value is greater than a minimum threshold userengagement value (e.g., a minimum number of active users and/ordownloads). In cases where the assignment conditions include a minimumuser engagement value, the record generation module 510 may refrain fromassigning function keywords to a search record 516 if the minimum userengagement value is not satisfied. This may prevent unpopularapplications from being assigned function keywords.

The record generation module 510 may assign additional data to thesearch records 516 along with the function keywords. For example, therecord generation module 510 may assign usage fraction values and/oruser engagement values to the search records 510 along with the functionkeywords. The record generation module 510 may assign this additionaldata to the search records 516 based on the assignment conditionsdescribed above. The search system owner may configure the recordgeneration module 510 to assign the additional data (e.g., which data toassign) and the conditions for assignment.

FIG. 13 illustrates a method for assigning function keywords and otherdata (e.g., usage fraction values and/or user engagement values) tosearch records 516. The method is directed to assigning functionkeywords and other data to a single search record. The record generationmodule 510 may repeat the method for each search record 516 in thesearch data store 512. The method is described with reference to thesearch system 110 and the UF determination system 108 of FIG. 5.

In block 1300, the record generation module 510 selects a search record516 for potential assignment of function keywords. In block 1302, therecord generation module 510 retrieves data from the UF determinationsystem 108 associated with the application of the search record 516. Forexample, the record generation module 510 may retrieve data from ananalytics record 524 (e.g., the usage fraction values and/or the userengagement values) for the application and a function mapping record 526(e.g., the function names and function metadata) for the application.

In block 1304, the record generation module 510 determines whetherassignment conditions are satisfied for the search record 516. If theassignment conditions are not satisfied, the record generation module510 may refrain from assigning function keywords and other data to thesearch record 516. If the assignment conditions are satisfied, therecord generation module 510 may assign the function keywords and otherdata (e.g., usage fraction values and/or user engagement values) to thesearch record 516 in block 1306. Note that the record generation module510 may update the function keywords and other data over time based onchanges in usage of the application.

Referring to FIG. 5, the search module 514 can use the search query 1104and the additional data included in the query wrapper 518 to generatethe search results 522. The search module 514 performs a search forsearch records 516 included in the search data store 512 in response tothe received query wrapper 518. In some implementations, the searchmodule 514 generates result scores 1114 for search records 516identified during the search. The result score 1114 associated with asearch record 516 may indicate the relevance of the search record 516 tothe search query 1104. A larger result score may indicate that thesearch record 516 is more relevant to the search query 1104. The searchmodule 514 may retrieve application download addresses 1106 from thescored search records 516. The search module 514 can transmit a resultscore 1114 along with one or more application download addresses 1106retrieved from a scored search record 516 in order to indicate the rankof the application download addresses among other transmittedapplication download addresses.

FIG. 14 illustrates an example search module 514 that includes a queryanalysis module 1400, a consideration set generation module 1402(hereinafter “set generation module 1402”), and a consideration setprocessing module 1404 (hereinafter “set processing module 1404”).

The query analysis module 1400 receives the query wrapper 518. The queryanalysis module 1400 analyzes the received search query 1104. Forexample, the query analysis module 1400 may perform various analysisoperations on the received search query 1104, such as tokenization ofthe search query 1104, filtering of the search query 1104, stemming,synonymization, and stop word removal. In some implementations, thequery analysis module 1400 may perform analysis targeted at identifyingquery terms associated with function names. In these implementations,the query analysis module 1400 may indicate whether the search query1104 includes query terms associated with function names and/or mayindicate the function names associated with the query terms. Forexample, if the query analysis module 1400 receives a search query“transit maps,” the query analysis module 1400 may identify the terms“transit” and/or “maps” as being associated with the “Navigate to alocation” function name. In some cases, a determination that query termsare associated with function names may be used by the set generationmodule 1402 and/or the set processing module 1404 for scoring (e.g., aspart of a query scoring feature indicating that query terms areassociated with certain function names).

The set generation module 1402 identifies a plurality of search records516 based on the output of the query analysis module 1400 (e.g., searchquery tokens). For example, the set generation module 1402 may identifya plurality of search records 516 based on matches between terms of thesearch query 1104 and terms in the application attributes 1204. In oneexample, the set generation module 1402 may identify a plurality ofsearch records 516 based on matches between terms of the search query1104 and terms included in the function keywords (e.g., function namesand/or function metadata). Inclusion of the function keywords may resultin the surfacing of search records 516 that may not otherwise have beenfound during the search. Additionally, or alternatively, the setgeneration module 1402 may identify a plurality of search records 516based on matches between terms of the search query 1104 and other termsincluded in the application attributes 1204, such as terms included in adescription of the application, the developer of the application, thegenre of the application, and/or reviews of the application.

In some implementations, the set generation module 1402 may calculate apreliminary score for the identified search records 516 of theconsideration set. In these implementations, the set generation module1402 may calculate the preliminary scores based on which fields of thesearch records 516 matched terms of the search query 1104. Differentfields of the search records 516 may have different weightings for thepurposes of calculating the preliminary scores. For example, thefunction keyword field (e.g., the function name and/or functionmetadata) may have a first weighting for the purposes of calculating apreliminary score while the other fields of the application attributes1204 (e.g., the description, genre, etc.) have different weightings forthe purposes of calculating a preliminary score. In examples where theset generation module 1402 calculates a preliminary score for a searchrecord 516, the set generation module 1402 may calculate the preliminaryscore for the search record 516 based on which fields include terms thatmatch terms of the search query 1104. For example, the set generationmodule 1402 may boost a preliminary score if one or more terms from thefunction keyword field matches one or more terms of the search query1104.

In some implementations, the set generation module 1402 may calculatethe preliminary score based on the usage fraction value associated withthe function name and/or function metadata that matches the search query1104. For example, the usage fraction value may be used as a weighting(e.g., a boosting value) for the purposes of calculating the preliminaryscore. In some implementations, the set generation module 1402 maycalculate the preliminary score based on the user engagement value andthe usage fraction value associated with the function name and/orfunction metadata that matches the search query 1104. For example, thenormalized usage fraction value may be used as a weighting (e.g., aboosting value) for the purposes of calculating the preliminary score.

The set processing module 1404 may score the search records in theconsideration set in order to generate a set of search results 522. Thescores associated with the search records 516 may be referred to as“result scores.” The set processing module 1404 may determine a resultscore for each of the search records in the consideration set. Theresult scores associated with a search record may indicate the relativerank of the search record (e.g., the application download addresses)among other search records. When ordering search results 522 (e.g., atthe set processing module 1404 and/or the user device 520), the searchresults 522 may be ordered from the highest result score to the lowestresult score from the top of the display to the bottom.

The set processing module 1404 may generate result scores 1114 forsearch records 516 in a variety of different ways. For example, asdescribed herein, the set processing module 1404 may generate resultscores 1114 for search records 516 based on function keywords (e.g.,function metadata and/or function names), usage fraction values 1204-3,user engagement values 1204-4, and/or other data included in theapplication attributes 1204.

In some implementations, the set processing module 1404 generates aresult score 1114 for a search record 516 based on one or more scoringfeatures. The scoring features may be associated with the search record516 and/or the search query 1104. A search record scoring feature(hereinafter “record scoring feature”) may be based on any dataassociated with a search record 516. For example, record scoringfeatures may be based on any data included in the application attributes1204, such as data indicating the popularity of an application (e.g.,number of downloads), the ratings (e.g., number of stars) associatedwith an application, and other application statistics.

A query scoring feature may include any data associated with the searchquery 1104. For example, query scoring features may include, but are notlimited to, a number of words in the search query 1104, the popularityof the search query 1104, and the expected frequency of the words in thesearch query 1104. As described above, the identification of query termsthat are associated with function names may be used as a query scoringfeature. For example, the query scoring feature may be a binary feature(e.g., 0/1) indicating whether the query terms are associated with afunction name, as determined by the query analysis module 1400.

A record-query scoring feature may include any data generated based ondata associated with both the search record 516 and the search query1104 that resulted in identification of the search record 516 by the setgeneration module 1402. For example, record-query scoring features mayinclude, but are not limited to, parameters that indicate how well theterms of the search query 1104 match the terms of the applicationattributes 1202, such as the function keywords (e.g., the function name1204 and/or the function metadata 1206). In some implementations,record-query scoring features may include the preliminary scorescalculated by the set generation module 1402. The record-query scoringfeatures may also include binary features (e.g., 0/1) indicating whetherterms of the search query 1104 match terms of the function keywordsand/or whether terms of the search query 1104 match function keywordsassociated with a dominant/specialized function. In someimplementations, record-query features (e.g., binary features) can beweighted by the usage fraction values (e.g., normalized usage fractionvalues) associated with the matched function keywords. The record-queryscoring features may also include a feature indicating how many termsfrom the search query match the function keywords. In someimplementations, a record-query scoring feature may be an indication ofthe coverage of the function keywords by the search query (e.g., thecoverage indicating a percentage of function keywords included in thequery terms).

The set processing module 1404 may generate a result score 1114 for asearch record 516 based on at least one of the record scoring features,the query scoring features, and the record-query scoring features. Forexample, the set processing module 1404 may determine a result score1114 based on one or more of the scoring features listed herein and/oradditional scoring features not explicitly listed. In some examples, theset processing module 1404 may include one or more machine learnedmodels (e.g., a supervised learning model) configured to receive one ormore scoring features. The one or more machine learned models maygenerate result scores 1114 based on at least one of the record scoringfeatures, the query scoring features, and the record-query scoringfeatures. For example, the set processing module 1404 may pair thesearch query 1104 with each search record 516 and calculate a vector offeatures for each (query, record) pair. The vector of features mayinclude one or more record scoring features, one or more query scoringfeatures, and one or more record-query scoring features. The setprocessing module 1404 may then input the vector of features into amachine-learned regression model to calculate a result score 1114 forthe search record. In some examples, the machine-learned regressionmodel may include a set of decision trees (e.g., gradient boosteddecision trees). In another example, the machine-learned regressionmodel may include a logistic probability formula. In some examples, themachine learned task can be framed as a semi-supervised learning task,where a minority of the training data is labeled with human curatedscores and the rest are used without human labels.

FIG. 15 illustrates an example method for performing a search accordingto the present disclosure. The method is described with reference to thesearch module 514 of FIG. 14. In block 1500, the query analysis module1400 receives a query wrapper 518 from a user device 520 and analyzesthe search query 1104. In block 1502, the set generation module 1402identifies a consideration set of search records based on the querywrapper 518 and the set processing module 1404 scores the search recordsof the consideration set. Put another way, the set processing module1404 may generate result scores 1114 for the search records of theconsideration set. For example, the set processing module 1404 maygenerate result scores 1114 for the search records based on at least oneof the assigned function metadata, function names, usage fractionvalues, and the user engagement values included in the search record,along with other scoring features. In block 1504, the set processingmodule 1404 generates search results 522 based on the scored searchrecords. The search results 522 may include application downloadaddresses 1106 and link data 1116 from the scored search records, alongwith result scores 1114 corresponding to the search records. In block1506, the set processing module 1404 transmits search results 522 to theuser device 520.

Although the function names described herein may be human readablefunction names that indicate the functionality provided by differentapplication (e.g., application states), in some implementations,function names may include, or be replaced by, identifiers that may notbe human readable. Such identifiers may be used in place of the functionnames. For example, event analytics data, function analytics data, usagefraction values, and user engagement values may be calculated based onthe unique identifiers in a similar manner described above. Furthermore,the search system 110 of the present disclosure may receive a querywrapper and generate search results in a similar manner described abovewhen such identifiers are used.

Modules and data stores included in the systems (e.g., 108, 110)represent features that may be included in the systems of the presentdisclosure. The modules and data stores described herein may be embodiedby electronic hardware, software, firmware, or any combination thereof.Depiction of different features as separate modules and data stores doesnot necessarily imply whether the modules and data stores are embodiedby common or separate electronic hardware or software components. Insome implementations, the features associated with the one or moremodules and data stores depicted herein may be realized by commonelectronic hardware and software components. In some implementations,the features associated with the one or more modules and data storesdepicted herein may be realized by separate electronic hardware andsoftware components.

The modules and data stores may be embodied by electronic hardware andsoftware components including, but not limited to, one or moreprocessing units, one or more memory components, one or moreinput/output (I/O) components, and interconnect components. Interconnectcomponents may be configured to provide communication between the one ormore processing units, the one or more memory components, and the one ormore I/O components. For example, the interconnect components mayinclude one or more buses that are configured to transfer data betweenelectronic components. The interconnect components may also includecontrol circuits (e.g., a memory controller and/or an I/O controller)that are configured to control communication between electroniccomponents.

The one or more processing units may include one or more centralprocessing units (CPUs), graphics processing units (GPUs), digitalsignal processing units (DSPs), or other processing units. The one ormore processing units may be configured to communicate with memorycomponents and I/O components. For example, the one or more processingunits may be configured to communicate with memory components and I/Ocomponents via the interconnect components.

A memory component may include any volatile or non-volatile media. Forexample, memory may include, but is not limited to, electrical media,magnetic media, and/or optical media, such as a random access memory(RAM), read-only memory (ROM), non-volatile RAM (NVRAM),electrically-erasable programmable ROM (EEPROM), Flash memory, hard diskdrives (HDD), magnetic tape drives, optical storage technology (e.g.,compact disc, digital versatile disc, and/or Blu-ray Disc), or any othermemory components.

Memory components may include (e.g., store) data described herein. Forexample, the memory components may include the data (e.g., records)included in the data stores. Memory components may also includeinstructions that may be executed by one or more processing units. Forexample, memory may include computer-readable instructions that, whenexecuted by one or more processing units, cause the one or moreprocessing units to perform the various functions attributed to themodules and data stores described herein.

The I/O components may refer to electronic hardware and software thatprovides communication with a variety of different devices. For example,the I/O components may provide communication between other devices andthe one or more processing units and memory components. In someexamples, the I/O components may be configured to communicate with acomputer network. For example, the I/O components may be configured toexchange data over a computer network using a variety of differentphysical connections, wireless connections, and protocols. The I/Ocomponents may include, but are not limited to, network interfacecomponents (e.g., a network interface controller), repeaters, networkbridges, network switches, routers, and firewalls. In some examples, theI/O components may include hardware and software that is configured tocommunicate with various human interface devices, including, but notlimited to, display screens, keyboards, pointer devices (e.g., a mouse),touchscreens, speakers, and microphones. In some examples, the I/Ocomponents may include hardware and software that is configured tocommunicate with additional devices, such as external memory (e.g.,external HDDs).

In some implementations, the systems may include one or more computingdevices (e.g., a computer search system) that are configured toimplement the techniques described herein. Put another way, the featuresattributed to the modules and data stores described herein may beimplemented by one or more computing devices. Each of the one or morecomputing devices may include any combination of electronic hardware,software, and/or firmware described above. For example, each of the oneor more computing devices may include any combination of processingunits, memory components, I/O components, and interconnect componentsdescribed above. The one or more computing devices of the systems mayalso include various human interface devices, including, but not limitedto, display screens, keyboards, pointing devices (e.g., a mouse),touchscreens, speakers, and microphones. The computing devices may alsobe configured to communicate with additional devices, such as externalmemory (e.g., external HDDs).

The one or more computing devices of the systems may be configured tocommunicate with the network 112. The one or more computing devices ofthe systems may also be configured to communicate with one another(e.g., via a computer network). In some examples, the one or morecomputing devices of the systems may include one or more servercomputing devices configured to communicate with user devices (e.g.,receive query wrappers and transmit search results), gather data fromdata source, index data, store the data, and store other documents. Theone or more computing devices may reside within a single machine at asingle geographic location in some examples. In other examples, the oneor more computing devices may reside within multiple machines at asingle geographic location. In still other examples, the one or morecomputing devices of the systems may be distributed across a number ofgeographic locations.

1. A method comprising: retrieving, at one or more computing devices,event analytics data for a software application installed on a pluralityof user devices, wherein the software application includes a pluralityof accessible application states, and wherein the event analytics dataindicates an amount of time each of the accessible application stateswas accessed by the user devices; mapping function names to theapplication states accessed by the user devices, wherein each of thefunction names indicates a functionality associated with the softwareapplication; for each of the function names, determining an amount oftime associated with the function name based on the amount of timeassociated with the application states to which the function name ismapped; determining a usage fraction value for each of the functionnames based on the amount of time associated with the function names,the usage fraction value for each function name indicating the amount oftime associated with the function name relative to the amount of timeassociated with each of the function names; assigning function keywordsto an application search record associated with the software applicationbased on the usage fraction values associated with the function names ofthe software application; receiving a search query from a querying userdevice; identifying the application search record based on a matchbetween terms of the search query and terms of the function keywordsassigned to the application search record; generating search resultsbased on the identified application search record; and transmitting thesearch results to the querying user device.
 2. The method of claim 1,wherein the function keywords include one or more of the function names.3. The method of claim 1, wherein the function keywords include functionmetadata that includes a plurality of terms that are associated with oneor more of the function names.
 4. The method of claim 1, wherein each ofthe function names is associated with a different set of functionkeywords and a different usage fraction value.
 5. The method of claim 4,wherein assigning the function keywords to the application search recordcomprises: determining whether the usage fraction value associated withthe function keywords is greater than a threshold value; and assigningthe function keywords to the application search record when the usagefraction value is greater than the threshold value.
 6. The method ofclaim 4, wherein assigning the function keywords to the applicationsearch record comprises: determining whether the usage fraction valueassociated with the function keywords is the greatest usage fractionvalue among the usage fraction values associated with the functionnames; and assigning the function keywords to the application searchrecord when the usage fraction value is the greatest usage fractionvalue.
 7. The method of claim 4, wherein generating the search resultscomprises scoring the application search record based on the usagefraction value associated with the function keywords.
 8. The method ofclaim 1, wherein generating the search results comprises scoring theapplication search record based on the match between the terms of thesearch query and the terms of the function keywords.
 9. The method ofclaim 1, wherein generating the search results comprises generating asingle search result for the identified application search record, thesingle search result including an application download addressconfigured to set the querying user device into a state that provides apreview of the software application and/or provides the softwareapplication for download.
 10. The method of claim 1, wherein generatingthe search results comprises: determining the number of user devicesfrom which the event analytics data was retrieved; and scoring theidentified application search record based on the number of userdevices.
 11. A system comprising: one or more memory componentsconfigured to store computer-readable instructions; and one or morecomputing devices including one or more processing units that executethe computer-readable instructions, the computer-readable instructions,when executed by the one or more processing units, cause the one or morecomputing devices to: retrieve event analytics data for a softwareapplication installed on a plurality of user devices, wherein thesoftware application includes a plurality of accessible applicationstates, and wherein the event analytics data indicates an amount of timeeach of the accessible application states was accessed by the userdevices; map function names to the application states accessed by theuser devices, wherein each of the function names indicates afunctionality associated with the software application; for each of thefunction names, determine an amount of time associated with the functionname based on the amount of time associated with the application statesto which the function name is mapped; determine a usage fraction valuefor each of the function names based on the amount of time associatedwith the function names, the usage fraction value for each function nameindicating the amount of time associated with the function name relativeto the amount of time associated with each of the function names; assignfunction keywords to an application search record associated with thesoftware application based on the usage fraction values associated withthe function names of the software application; receive a search queryfrom a querying user device; identify the application search recordbased on a match between terms of the search query and terms of thefunction keywords assigned to the application search record; generatesearch results based on the identified application search record; andtransmit the search results to the querying user device.
 12. The systemof claim 11, wherein the function keywords include one or more of thefunction names.
 13. The system of claim 11, wherein the functionkeywords include function metadata that includes a plurality of termsthat are associated with one or more of the function names.
 14. Thesystem of claim 11, wherein each of the function names is associatedwith a different set of function keywords and a different usage fractionvalue.
 15. The system of claim 14, wherein the one or more computingdevices are configured to assign the function keywords to theapplication search record by: determining whether the usage fractionvalue associated with the function keywords is greater than a thresholdvalue; and assigning the function keywords to the application searchrecord when the usage fraction value is greater than the thresholdvalue.
 16. The system of claim 14, wherein the one or more computingdevices are configured to assign the function keywords to theapplication search record by: determining whether the usage fractionvalue associated with the function keywords is the greatest usagefraction value among the usage fraction values associated with thefunction names; and assigning the function keywords to the applicationsearch record when the usage fraction value is the greatest usagefraction value.
 17. The system of claim 14, wherein the one or morecomputing devices are configured to generate the search results byscoring the application search record based on the usage fraction valueassociated with the function keywords.
 18. The system of claim 11,wherein the one or more computing devices are configured to generate thesearch results by scoring the application search record based on thematch between the terms of the search query and the terms of thefunction keywords.
 19. The system of claim 11, wherein the one or morecomputing devices are configured to generate the search results bygenerating a single search result for the identified application searchrecord, the single search result including an application downloadaddress configured to set the querying user device into a state thatprovides a preview of the software application and/or provides thesoftware application for download.
 20. The system of claim 11, whereinthe one or more computing devices are configured to generate the searchresults by: determining the number of user devices from which the eventanalytics data was retrieved; and scoring the identified applicationsearch record based on the number of user devices.